Color balance correction based on skin color and highlight color

ABSTRACT

An image processing apparatus for applying a color balance correction to input image data, comprises a detection unit which detects an object included in the image data; an object color calculation unit which calculates an object color as a color of the object; an acquisition unit which acquires a highlight color in the image data; a calculation unit which calculates a correction value for the object color and the highlight color based on a relative positional relationship between the object color and the highlight color on a color space; and a color balance correction unit which applies the color balance correction to the highlight color and the object color using the correction value.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus, imageprocessing method, and computer-readable medium which correct a colorbalance of input digital image data.

2. Description of the Related Art

Conventionally, a method of estimating a light source at the time ofshooting of image data shot by a digital camera or that obtained byoptically scanning a film shot by an analog camera, and applying a colorbalance correction to the image data has been proposed.

Japanese Patent Laid-Open No. 2002-152772 describes the followingmethod. That is, a blackbody locus of gray and human skin color pixelsbased on a representative spectral distribution of a camera is held inadvance. RGB components of respective pixels are multiplied by variouscoefficients so as to attain inverse conversions of a color balancecorrection of a camera, and the number of pixels after conversion, whichare located adjacent to the blackbody locus, is evaluated, therebyestimating a light source at the time of shooting. After the lightsource is estimated, the converted image is converted into colors undera predetermined light source.

The aforementioned color balance correction method suffers the followingproblem. That is, image data shot by digital cameras and those obtainedby optically scanning films shot by analog cameras include a widevariety of image data, and a gray region and skin color region may beerroneously detected. For example, assuming a portrait photo of a personwho holds a yellow flower bouquet in image data, a skin color can becorrectly detected by human face detection, but a yellow flower regionmay often be erroneously detected as a highlight color. In this case,since a gray region in an image is erroneously detected, an estimationerror occurs with a high possibility upon estimation of a light sourceat the time of shooting.

On the other hand, in face detection upon analyzing image data, forexample, if a mountain surface in a landscape photo is erroneouslydetected as a face, a skin color is erroneously detected. In this case,a highlight color can be correctly estimated, but a light source at thetime of shooting is erroneously estimated with a high possibility due toerroneous detection of the skin color.

When the color balance correction is applied using the above lightsource estimation value, a color balance of image data is largelybroken, thus causing image adverse effects.

SUMMARY OF THE INVENTION

According to one aspect of the present invention, there is provided animage processing apparatus for applying a color balance correction toinput image data, comprising: a detection unit which detects an objectincluded in the image data; an object color calculation unit whichcalculates an object color as a color of the object; an acquisition unitwhich acquires a highlight color in the image data; a calculation unitwhich calculates a correction value for the object color and thehighlight color based on a relative positional relationship between theobject color and the highlight color on a color space; and a colorbalance correction unit which applies the color balance correction tothe highlight color and the object color using the correction value,wherein the calculation unit calculates the correction value so that atleast one of a length and a direction of a vector formed by thehighlight color and the object color approaches a predeterminedreference vector on the color space by applying the correction value tothe highlight color and the object color.

According to another aspect of the present invention, there is providedan image processing apparatus for applying a color balance correction toinput image data, comprising: a detection unit which detects an objectincluded in the image data; a calculation unit which calculates anobject color as a color of the object; an acquisition unit whichacquires a highlight color in the image data; a reliability calculationunit which calculates a reliability for a value of the object colorcalculated by the calculation unit and a reliability for a value of thehighlight color acquired by the acquisition unit based on a relativepositional relationship between the object color and the highlight coloron a color space, and differences between the object color and thehighlight color, and reference values; and a color balance correctionunit which applies the color balance correction to the object color andthe highlight color by controlling a correction amount for the objectcolor and the highlight color using the reliability for the value of theobject color and the reliability for the value of the highlight color.

According to another aspect of the present invention, there is providedan image processing method for applying a color balance correction toinput image data, comprising: a detection step of detecting an objectincluded in the image data; an object color calculation step ofcalculating an object color as a color of the object; an acquisitionstep of acquiring a highlight color in the image data; a calculationstep of calculating a correction value for the object color and thehighlight color based on a relative positional relationship between theobject color and the highlight color on a color space; and a colorbalance correction step of applying the color balance correction to thehighlight color and the object color using the correction value, whereinin the calculation step, the correction value is calculated so that atleast one of a length and a direction of a vector formed by thehighlight color and the object color approaches a predeterminedreference vector on the color space by applying the correction value tothe highlight color and the object color.

According to another aspect of the present invention, there is providedan image processing method for applying a color balance correction toinput image data, comprising: a detection step of detecting an objectincluded in the image data; a calculation step of calculating an objectcolor as a color of the object; an acquisition step of acquiring ahighlight color in the image data; a reliability calculation step ofcalculating a reliability for a value of the object color calculated inthe calculation step and a reliability for a value of the highlightcolor acquired in the acquisition step based on a relative positionalrelationship between the object color and the highlight color on a colorspace, and differences between the object color and the highlight color,and reference values; and a color balance correction step of applyingthe color balance correction to the object color and the highlight colorby controlling a correction amount for the object color and thehighlight color using the reliability for the value of the object colorand the reliability for the value of the highlight color.

According to the present invention, image adverse effects described inthe related art can be suppressed. That is, when a relative positionalrelationship between a highlight color and object color (for example,human skin color) extracted from image data deviates from that in aproper case, it is judged that at least one of the highlight color andobject color is wrong, and a low reliability is set for the extractedcolor information. By suppressing a correction amount at the time of thecolor balance correction using this reliability, the image adverseeffects described in the related art can be suppressed. Hence, using theimage processing apparatus and method of the present invention, a stablecolor balance correction without any serious adverse effects can beimplemented.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the hardware arrangement required toexecute an image processing method according to the present invention;

FIGS. 2A and 2B are functional block diagrams according to the firstembodiment;

FIG. 3 is a flowchart of HL color calculation processing in an imageanalysis unit according to the present invention;

FIG. 4 is a graph for explaining processing for determining a highlightluminance value;

FIGS. 5A and 5B are graphs for explaining processing for modifying an HLcolor of an input image using an HL color blackbody locus;

FIGS. 6A, 6B, 6C, and 6D are graphs for explaining processing forcalculating a reliability using an HL color blackbody locus;

FIGS. 7A, 7B, and 7C are graphs for explaining processing for modifyingan HL color of an input image using a skin color and the HL color;

FIG. 8 is a flowchart of light source estimation processing according tothe present invention;

FIGS. 9A and 9B are graphs showing a setting example of a gray axis in alight source estimation unit according to the present invention;

FIG. 10 is a flowchart of gray/skin color map generation processing inthe light source estimation unit according to the present invention;

FIG. 11 is a graph for explaining processing for specifying skin colorpixels according to the present invention;

FIG. 12 is a flowchart of a second color conversion unit according tothe first embodiment;

FIG. 13 is a table showing a holding example of white point informationunder respective light sources according to the present invention;

FIGS. 14A and 14B are graphs for explaining processing for determining afinal white point according to the present invention;

FIG. 15 is a graph for explaining conversion amounts before and aftersecond color conversion according to the present invention;

FIGS. 16A and 16B are graphs for explaining an image analysis resultmodification unit according to the second and third embodiments;

FIG. 17 is a flowchart of processing for determining a final white pointaccording to the fifth embodiment;

FIGS. 18A and 18B are graphs of processing for determining a final whitepoint according to the fifth embodiment;

FIG. 19 is a functional block diagram according to the sixth embodiment;

FIG. 20 is a flowchart of processing for calculating a reliabilityaccording to the sixth embodiment;

FIG. 21 is a functional block diagram according to the seventhembodiment;

FIG. 22 is a flowchart of a 3D LUT generation unit according to theseventh embodiment; and

FIG. 23 is a functional block diagram according to the seventhembodiment.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

A first embodiment of the present invention will be described below.Note that the following description will be given under the assumptionof image processing inside a printer main body. However, this is merelyan embodiment, and the present invention is not limited to the followingembodiments.

(Hardware Arrangement)

FIG. 1 shows the hardware arrangement which can implement an imageprocessing method according to the present invention. Note that FIG. 1shows an example in this embodiment, and the present invention is notlimited to the arrangement shown in FIG. 1. Referring to FIG. 1, animage processing apparatus 100 has a CPU 101, RAM 102, and ROM 103, andrespective functional blocks are connected to a system bus 112. Programcodes that implement the operations of the present invention are storedin the ROM 103. These program codes are loaded onto the RAM 102 and areexecuted by the CPU 101 when the corresponding operations are performed.

The image processing apparatus 100 has a secondary storage device 104such as an HDD or flash memory, and a mouse 108 and keyboard 109 areconnected via an interface 105. Also, a printer 110 is connected via aninterface 106, and a monitor 111 used to evaluate images before andafter correction is connected via a video card 107.

The arrangement of the image processing apparatus shown in FIG. 1 is,for example, that of a PC (personal computer) as a general informationprocessing apparatus.

[Overview of Overall Sequence]

The overall configuration of image processing according to the presentinvention will be described below with reference to FIG. 2A. In thisembodiment, an input image 200 can be a digital image shot by a digitalcamera or that obtained by optically scanning a film shot using ananalog camera. The format of input image data is not particularlylimited as long as the present invention is applicable to the imagedata. In this embodiment, assume that a color space of an input image issRGB (standard RGB) color space, and each pixel value is configured by8-bit R, G, and B component values, for the sake of simplicity.

The input image 200 is input to an image reduction unit 201, whichgenerates a reduced image required for an image analysis unit 202 (to bedescribed later) to execute processing. Using the reduced image, theprocessing load on the image analysis unit 202 can be reduced comparedto a case in which the input image 200 is input intact to the imageanalysis unit 202. A reduction processing method in this unit is notparticularly limited.

The reduced image is input to the image analysis unit 202, and undergoesimage analysis processing. The image analysis processing includesprocessing for extracting an object (for example, a human face) in animage, and calculating an object color (a human skin color when theobject is a human face), and processing for measuring a histogram of theentire image and calculating a highlight color (to be referred to as anHL color hereinafter). In this case, the HL color is the color of thebrightest region of an image. Furthermore, image feature amounts to becalculated include average values, maximum values, and minimum values oflightness and saturation values of the entire image.

A plurality of types of image feature amounts obtained as analysisresults of the image analysis unit 202 are input to an image analysisresult modification unit 203, which modifies the image analysis results.Analysis values obtained as the analysis results of the image analysisunit 202 are not always correct. For example, as for human facedetection, a non-human face region may be erroneously detected as a faceregion at a given probability. A color balance correction (to bedescribed later) is performed based on the calculated HL color of animage. In this case, if a color different from a light source color atthe time of shooting is erroneously calculated as the HL color, thecolor balance correction often does not normally operate. Inconsideration of the above problems, in this embodiment, the imageanalysis result modification unit 203 modifies respective featureamounts by various checking processes, and image processing in thesubsequent stage is executed using the modified values, thus outputtinga stable image which suffers less adverse image effects.

Next, a light source estimation unit 204 estimates a light source at thetime of shooting using the HL color modified by the image analysisresult modification unit 203 and the object color. The present inventionis not particularly limited to the estimation method of the light sourceat the time of shooting, and an arbitrary method disclosed in, forexample, Japanese Patent Laid-Open No. 2002-152772 described in therelated art may be used. A color temperature under the light source atthe time of shooting, which is estimated by the light source estimationunit 204, is calculated, and a highlight color under that condition isacquired, thus achieving first highlight color calculation processing.

When the light source estimation unit 204 performs light sourceestimation at the time of shooting, for example, a color temperaturevalue of the light source at the time of shooting is determined as aresult of the estimation, and is input to a first color conversion unit205. Normally, since an image shot by a digital camera has undergone aunique color balance correction inside the camera, the first colorconversion unit 205 cancels the color balance correction applied insidethe camera to convert the input image 200 to a state before the colorbalance correction. In this case, by canceling the color balancecorrection unique to an image capturing device (for example, a camera)applied to the input image, an image before the color balance correctioncan be acquired.

The image after first color conversion in the first color conversionunit 205 is input to a second color conversion unit 206, which appliessecond color conversion processing to that image. The second colorconversion unit 206 applies color conversion processing to the image ina state in which the color balance correction inside the camera iscanceled in consideration of human visual characteristics, so as toapproximate the actual “appearance” under the light source at the timeof shooting. The image after the second color conversion is stored inthe secondary storage device 104, is displayed on the monitor 111, or isprinted on a print medium by the printer 110 as an output image 207. Theblock diagram of the processing of this embodiment has been described.

[Operations of Respective Processing Units]

Details of the processing will be described below with reference to theflowchart and the like in association with the operations of therespective units shown in FIG. 2A.

(Image Reduction Unit 201)

The image reduction unit 201 reduces the input image 200 to an imagehaving a predetermined resolution, so as to improve the efficiency ofthe processing of the subsequent image analysis unit 202. For example,when an input image is a relatively high-resolution image having4000×3000 pixels, if lightness and saturation values are calculated for,for example, all pixels at that resolution, a huge processing time isrequired for the image analysis unit 202. Hence, in this embodiment, theinput image is reduced to an image having a minimum resolution (forexample, 640×480 pixels) required for the image analysis unit 202, andthis reduced image is input to the image analysis unit 202.

Note that in this embodiment, the reduction method is not particularlylimited, and either of a nearest neighbor reduction method and a linearinterpolation method may be used. The number of types of reduced imagesgenerated by the image reduction unit 201 is not limited to one. Forexample, a resolution of about 640×480 pixels suffices as the imageresolution required to calculate average lightness and averagesaturation values in the image analysis unit 202. However, when objectdetection such as face detection is to be executed, a resolution ofabout 1024×768 pixels is required. In this case, the image reductionunit 201 may generate two types of reduced images, and may input them tothe image analysis unit 202.

(Image Analysis Unit 202)

FIG. 2B is a block diagram showing detailed processing of the imageanalysis unit 202 shown in FIG. 2A. In this embodiment, the imageanalysis unit 202 is configured by only three blocks for the sake ofsimplicity, but the present invention is not limited to this. As shownin FIG. 2B, an HL color determination unit 301 calculates an HL color ofan image based on a reduced image 300 obtained by the image reductionunit 201. The reduced image 300 is also input to an object detectionunit 302, which detects a human face region as a principal object in theimage. The principal object in this case indicates a main target to beshot in the shot image. In this embodiment, since an image includes aperson as a main object, a principal object is a face of that person.Furthermore, an object color calculation unit 303 calculates an objectcolor, that is, a skin color based on position information of the faceregion.

(Object Detection Unit 302, Object Color Calculation Unit 303)

The object detection unit 302 executes processing for detecting aprincipal object in an input image (in this case, the reduced image300). The principal object mainly includes a human face. As a human facedetection method used at this time, various methods have already beenproposed. Japanese Patent Laid-Open No. 2002-183731 discloses thefollowing method. That is, eye regions are detected from an input image,and a region around the eye regions is determined as a face candidateregion. Luminance gradients and weights for luminance gradients arecalculated for respective pixels, and are compared with gradients andweights for gradients of an ideal face reference image, which is set inadvance. When an average angle between gradients is equal to or smallerthan a predetermined threshold, it is determined that the input imagehas a face region.

According to Japanese Patent Laid-Open No. 2003-30667, a skin colorregion is detected from an image, and human iris color pixels aredetected from the detected region, thereby detecting the positions ofeyes.

Furthermore, according to Japanese Patent Laid-Open No. 8-63597, degreesof matching between a plurality of templates having face shapes and animage are calculated. A template corresponding to the highest degree ofmatching is selected, and when the highest degree of matching is equalto or higher than a predetermined threshold, a region in the selectedtemplate is set as a face candidate region. Using this template, thepositions of eyes can be detected.

Moreover, according to Japanese Patent Laid-Open No. 2000-105829, anentire image or a designated region in the image is scanned using a noseimage pattern as a template, and a best matching position is output as anose position. Next, a region above the nose position of the image isconsidered as that including eyes, and the eye existing region isscanned using an eye image pattern as a template to calculate degrees ofmatching. Then, an eye existing candidate position set as a set ofpixels having degrees of matching larger than a certain threshold iscalculated. Furthermore, continuous regions included in the eye existingcandidate position set are divided as clusters, and distances betweenthe clusters and the nose position are calculated. The clusters havingthe shortest distance are determined as those including eyes, thusallowing detection of organ positions.

In addition, as methods of detecting a face and organ positions,Japanese Patent Laid-Open Nos. 8-77334, 2001-216515, 5-197793, 11-53525,2000-132688, 2000-235648, and 11-250267 are known. Furthermore, a largenumber of methods such as Japanese Patent No. 2541688 have beenproposed. In this embodiment, the method to be used is not particularlylimited.

Furthermore, in these various face detection methods, a face detectionreliability R_face can be generally calculated. This reliability R_facehas a value ranging from 0 to 1.0, and when the reliability R_face iscloser to 1.0, it is judged that the reliability of the detection resultis higher. That is, the reliability can be determined based on adifference from a reference value. As this face detection reliability,in case of face detection based on a template matching method, a degreeof matching with each template can be used as the face detectionreliability. Also, in case of a face detection method by comparing facefeature amounts such as the eye positions and skin color with thosewhich are held in advance, a degree of matching with these featureamounts can be used as the face detection reliability. In thisembodiment, any of these methods may be used.

When the object detection has succeeded, coordinates indicating theposition of the principal object can be obtained. The coordinates may bethose of a rectangle which includes a face region, or may be those of acircle, which is expressed by the center and radius. The coordinateinformation is input to the object color calculation unit 303, whichanalyzes colors of the principal object region. The color analysisgenerally means calculations of an average color in the detectedprincipal object region. When pixels which form an image are expressedby RGB values, the average color can be obtained by calculating averagevalues of RGB values of pixels included in the principal object region.

In this embodiment, an object color calculated by the above method ishandled as (ObjR, ObjG, ObjB). Also, values (ObjY, ObjCb, ObjCr)obtained by converting the object color onto a YCbCr color space aresimilarly handled as an object color. As another method, RGB values ofpixels included in the principal object region may be converted intoluminance and color difference components, and average values of thecolor difference components may be calculated. In this case, aluminance-color difference color space may be, for example, any of aYC1C2 space and CIE L*a*b* space.

When the object detection unit 302 detects a plurality of principalobjects, average colors calculated for respective object regions may befurther averaged. Alternatively, in order to enhance a correctioneffect, an object color farthest from an ideal object color, which isset in advance, may be used as a reference. Conversely, in order toweaken a correction effect, a principal object closest to the idealobject color may be used as a reference.

(Calculation of Image HL Color)

The HL color determination unit 301 shown in FIG. 2B calculates an HLcolor of an input image. The HL color determination unit 301 executessecond highlight color calculation processing. Various HL colorcalculation methods have been proposed, and any of these methods may beused as long as the present invention is applicable. In this embodiment,the HL color calculation method will be described below with referenceto the sequence shown in FIG. 3. Note that the sequence shown in FIG. 3is implemented when the CPU 101 reads out and executes a program storedin, for example, the RAM 102 as a storage unit.

Referring to FIG. 3, a histogram of the reduced image 300 is calculatedin steps S401 to S404. In step S401, the HL color determination unit 301converts RGB values of respective pixels in the reduced image 300 intoluminance and color difference components using known formulas. In thiscase, for example, RGB values are converted into YCbCr values. In stepS402, the HL color determination unit 301 calculates a frequencyhistogram of luminance components Y. In this case, the histogram can becalculated by incrementing a Y-th element of a HistY array, which isinitialized to zero.

In step S403, the HL color determination unit 301 makes the followingcalculations for arrays MeanCb and MeanCr so as to calculate averagevalues of color difference values Cb and Cr for respective luminancevalues. The calculations are given by:MeanCb[Y]+=CbMeanCr[Y]+=Cr  (1)

Upon completion of the processes in steps S401 to S404 for all thepixels of the reduced image 300, the luminance histogram array HistYstores a frequency distribution shown in FIG. 4. Hence, in step S405 theHL color determination unit 301 determines a highlight luminance valueHLY using HistY. As HLY in this embodiment, values of HistY[Y] areaccumulated in turn from smaller luminance values, and a luminance valueY when the accumulated frequency reaches 99% of the entire reduced image300 is determined as the highlight luminance value HLY. Note that theaccumulated frequency ratio required to determine HLY is not limited to99%. On the other hand, if a luminance value can be specified duringaccumulation processing, the luminance value may be determined at thattime, and the processing may end.

In step S406, the HL color determination unit 301 calculates colorvalues HLCb and HLCr of an HL region using the determined value HLY.These values are respectively calculated by:HLCb=MeanCb[HLY]÷HistY[HLY]HLCr=MeanCr[HLY]÷HistY[HLY]  (2)

With the aforementioned processing, the second highlight colordetermination processing is attained. The calculated object color and HLcolor (HLCb and HLCr values) are input to the image analysis resultmodification unit 203.

(Image Analysis Result Modification Unit 203)

The image analysis result modification unit 203 is a processing unitwhich modifies the analysis results calculated by the image analysisunit 202. This unit modifies the HL color of the input image calculatedby the image analysis unit 202 using an HL color blackbody locus basedon a spectral distribution, which is held in advance. The HL colorblackbody locus will be described first.

Assuming an ideal blackbody, a distribution of the wavelengths of lightradiated by the blackbody at a certain temperature can be derived. Forexample, when a color “white” is expressed by a blackbody temperature,if the temperature is low, the color is close to orange; if thetemperature is high, it is close to bluish white. The blackbodytemperature expressed in this case is called a color temperature (K:Kelvin). The HL color blackbody locus used in this embodiment representsa locus of colors of a bright region, that is, HL colors when the colortemperature of illumination is changed. For example, the blackbody locuswill be described below with reference to FIG. 5A.

FIG. 5A is a graph which plots, on a Cb-Cr plane, the color differencecomponent values HLCb and HLCr of the HL color of the input image andthe HL color blackbody locus based on the spectrum distribution, whichis held in advance. FIG. 5A shows HL colors when the color temperatureis changed from, for example, 3000K (●) to 7000K (●). A locus whichconnects HL colors obtained when the color temperature is changed is anHL color blackbody locus 601.

Referring to FIG. 5A, coordinates 602 (Δ) represent those of the colordifference component values HLCb and HLCr of the HL color analyzed fromthe input image. Also, a two-headed arrow 603 represents a distance to aposition where the coordinates 602 are closest to the HL color blackbodylocus 601. A broken arrow 604 represents a distance between thecoordinates 602 and (Cb, Cr)=(0, 0). As shown in FIG. 5A, an image,which has undergone a color balance correction inside a digital camera,generally has the color difference component values HLCb and HLCr of theHL color of the input image, which are located at a position close tothe HL color blackbody locus 601. In this case, since the detected HLcolor is more likely to be correctly detected, it is desired topositively execute the color balance correction. Conversely, when thecolor difference component values HLCb and HLCr of the HL color of theinput image are farther away from the HL color blackbody locus 601,since the HL color is more likely to be erroneously detected, it isdesired to suppress the correction amount of the color balancecorrection. That is, the correction amount is controlled according to adifference between a reference value and the analyzed value.

For example, when the user brings a camera closer to yellow flowers toshoot an image of the yellow flowers within the entire frame in a macromode, the yellow flower portion is unwantedly detected as an HL color.In this case, when this HL color is used intact to apply the colorbalance correction that sets the HL color to be white, the yellowflowers are also corrected to be white, thus causing a considerableadverse effect. An example in which the color difference componentvalues HLCb and HLCr of the HL color of the input image are farther awayfrom the HL color blackbody locus 601 will be described below withreference to FIG. 5B.

FIG. 5B is a graph which plots, on the color difference Cb-Cr plane, thecolor difference component values HLCb and HLCr of the HL color of theinput image and the blackbody locus of HL colors based on the spectraldistribution, which is held in advance. As in FIG. 5A, a locus thatconnects HL colors obtained when the color temperature is changed from3000K (●) to 7000K (●) is set as the HL color blackbody locus 601.

As an example in which the color difference component values HLCb andHLCr of the HL color of the input image are farther away from the HLcolor blackbody locus 601, these color difference component values HLCband HLCr are indicated by coordinates 702 (Δ). A two-headed arrow 703represents a distance to a position where the coordinates 702 areclosest to the HL color blackbody locus 601. Also, a broken arrow 704represents a distance between the coordinates 702 and (Cb, Cr)=(0, 0).In this embodiment, a method for suppressing the correction amount ofthe color balance correction is implemented by modifying the colordifference component values HLCb and HLCr of the HL color of the inputimage.

As a method of modifying the color difference component values HLCb andHLCr of the HL color of the input image, modification is made based onthe distance between the color difference component values HLCb and HLCrof the HL color of the input image and the HL color blackbody locus. Anexample in which the color difference component values HLCb and HLCr ofthe HL color of the input image are modified will be described belowwith reference to FIGS. 5A and 5B and FIG. 6A. FIG. 6A is a graphshowing a function which defines areliability R_dis of the colordifference component values HLCb and HLCr of the HL color of the inputimage with respect to the distance between the color differencecomponent values HLCb and HLCr of the HL color of the input image andthe HL color blackbody locus 601.

Referring to FIG. 6A, let L be a distance between the coordinates of thecolor difference component values HLCb and HLCr of the HL color of theinput image and the HL color blackbody locus 601. Also, let R_dis be areliability, which is calculated based on the distance L, of the colordifference component values HLCb and HLCr of the HL color analyzed fromthe input image. Furthermore, Lmax used upon calculation of thereliability R_dis is set in advance. Assume that Lmax is defined incorrespondence with characteristics (an object type, colors included inan image, etc.) of a target image. In this embodiment, assume that thereliability R_dis assumes a value ranging from 0 (minimum) to 1.0(maximum). In FIG. 6A, the reliability R_dis is calculated by:When LMax>L,R_dis=(LMax−L)/LMaxWhen LMax≦L,R_dis=0.0  (3)

For example, assume that the coordinates 602 of the color differencecomponent values HLCb and HLCr of the HL color of the input image shownin FIG. 5A are expressed by (HLCb, HLCr)=(−20, 20). Also, assume thatthe coordinates 702 of the color difference component values HLCb andHLCr of the HL color of the input image shown in FIG. 5B are expressedby (HLCb, HLCr)=(−20, 4).

The distance L between the color difference component values HLCb andHLCr of the HL color of the input image and the HL color blackbody locus601 is calculated first. As a method of calculating the distance L, forexample, a distance between the color difference component values HLCband HLCr of the HL color of the input image and a position correspondingto a normal depending from the coordinates of the color differencecomponent values to the HL color blackbody locus 601 is calculated asthe distance L.

For example, assume that the distance L between the coordinates 602 ofthe color difference component values HLCb and HLCr of the HL coloranalyzed from the input image and the HL color blackbody locus 601 inFIG. 5A is 10. Also, assume that the distance L between the coordinates702 of the color difference component values HLCb and HLCr of the HLcolor analyzed from the input image and the HL color blackbody locus 601in FIG. 5B is 50.

Next, a reliability R_dis is calculated using pre-set LMax and thecalculated distance L using equations (3). For example, the reliabilityR_dis is calculated to have pre-set LMax=100. In the followingdescription, pre-set LMax=100, but it is a value to be arbitrarily set,and is not particularly limited. Calculation examples of thereliabilities R_dis of the coordinates 602 in FIG. 5A and thecoordinates 702 in FIG. 5B are as follows.Reliability of HL color coordinates 602: R_dis=(100−10)/100=0.9Reliability of HL color coordinates 702: R_dis=(100−50)/100=0.5

Next, the coordinates of the color difference component values HLCb andHLCr of the HL color analyzed from the input image are converted usingthe calculated reliability R_dis. As a coordinate conversion method,processing for making the coordinates of the color difference componentvalues HLCb and HLCr of the HL color analyzed from the input image becloser to an origin (Cb, Cr)=(0, 0) as a reference position is executed.This coordinate conversion formula is given such that (Cb′, Cr′) arecoordinates after conversion:(Cb′,Cr′)=(HLCb×R_dis,HLCr×R_dis)  (4)

Examples of coordinate conversion processing for making the coordinates602 and 702 of the HL colors be closer to (Cb, Cr)=(0, 0) using thereliabilities R_dis with respect to the coordinates shown in FIGS. 5Aand 5B are as follows.Coordinates 602: (Cb′,Cr′)=(−20×0.9, 20×0.9)=(−18,18)Coordinates 702: (Cb′,Cr′)=(−20×0.5,4×0.5)=(−10,2)

Next, the color difference component values HLCb and HLCr of the HLcolor analyzed from the input image are replaced by the coordinatevalues after conversion:HLCb=Cb′HLCr=Cr′  (5)

In FIG. 5A, the coordinates 602 (HLCb, HLCr) of the color differencecomponents of the HL color of the input image are modified from (−20,20) to (−18, 18). Also, in FIG. 5B, the coordinates 702 (HLCb, HLCr) ofthe color difference components of the HL color of the input image aremodified from (−20, 4) to (−10, 2).

As described above, the color difference component values HLCb and HLCrof the HL color analyzed from the input image follow the relationshipbetween the reliability R_dis and distance L, which is defined in FIG.6A. That is, as the distance to the HL color blackbody locus 601 becomesshorter, as shown in FIG. 5A, the degree of modification is decreased,and as the distance to the HL color blackbody locus 601 becomes farther,as shown in FIG. 5B, the degree of modification is increased.

When larger modification is applied to the coordinates 702 shown in FIG.5B compared to coordinates having the smaller distance L, the HL colorof the image becomes closer to the origin. When light source estimation(to be described later) is executed using the HL color near the origin,a gray pixel group as a target of the light source estimation becomes anachromatic color at a higher probability. Even when the light sourceestimation is executed using a gray pixel group of the achromatic colorand a skin color pixel group, a light source estimation value output asa result of the light source estimation assumes a value around a colortemperature (5000 to 5500K) under daylight as a standard color at ahigher probability. Since the correction amount of the color balancecorrection becomes weak for an image which is determined that its HLcolor is near the color temperature, the color balance correction amount(to be described later) can be consequently suppressed by making the HLcolor be closer to the origin.

Therefore, for example, assuming that an input image includes an imageof yellow flowers shot in the macro mode, the HL color of the image isyellow having a high saturation value, and is farther away from the HLcolor blackbody locus. In this case, since the HL color is modified to aposition near the origin to suppress the color balance correction, acorrection error that the yellow flowers are corrected to be white canbe suppressed.

This embodiment has been described under the assumption that an inputimage has undergone the color balance correction in an image capturingapparatus at the time of shooting. The color balance correction in theimage capturing apparatus basically executes color balance control sothat the HL color of an image is located near the blackbody locus shownin FIG. 5A. For this reason, for an image shot by the image capturingapparatus in a normal operation, the color balance correctionsatisfactorily functions even when the present invention is applied, anda color balance correction effect is never reduced due to excessivesuppression.

However, a possibility that the color balance correction in the imagecapturing apparatus does not normally function due to an arbitrarycause, and the HL color of an image largely deviates from the blackbodylocus is not zero. For example, in a recent image capturing apparatus, ahuman face region is detected from a shooting field angle, and colorbalance control is executed using that face color. At this time, forexample, when a mountain surface in the landscape is erroneouslyrecognized as a human face, since a skin color value is erroneouslydetected, the color balance may be largely broken, and the HL color ofan image may largely deviate from the blackbody locus. When the presentinvention is applied to such an image, the color balance correction maybe suppressed, and an image having an improper color balance may beoutput intact. However, the probability of the aforementioned detectionerror is very low. Hence, as a characteristic feature of the presentinvention, larger importance is attached to a merit that the colorbalance of an yellow flower image is not erroneously corrected ratherthan a demerit that the aforementioned image having the improper colorbalance cannot be corrected.

In this embodiment, when the reliability R_dis is calculated, as shownin FIG. 6A, positions from a distance L=0 to LMax are coupled by astraight line, and the reliability R_dis decreases with increasingdistance L. Using FIG. 6A, between input images having slightlydifferent HL colors, since their reliabilities R_dis are not sodifferent, the color balance correction processing can be similarlyapplied to these images without generating a large difference. Forexample, when images of the same area are shot within a short period oftime like those which are shot in a continuous shooting mode, inputimages having slightly different HL colors are generated. Since thereliability R_dis is changed according to the distance L, as shown inFIG. 6A, images corrected results after the color balance correction canbe generated.

In this embodiment, the reliability R_dis is calculated using FIG. 6A.However, the present invention is not limited to this. For example,another reliability calculation method will be described using FIG. 6B.Referring to FIG. 6B, as in FIG. 6A, let L be a distance between thecoordinates of the color difference component values HLCb and HLCr ofthe HL color analyzed from the input image and the HL color blackbodylocus. Also, let R_dis be a reliability, which is calculated based onthe distance L, of the color difference component values HLCb and HLCrof the HL color analyzed from the input image. Furthermore, Lmax usedupon calculation of the reliability R_dis is set in advance. In thisembodiment, assume that the reliability R_dis assumes a value rangingfrom 0 (minimum) to 1.0 (maximum).

As a difference from FIG. 6A, in FIG. 6B, the reliability R_dis is setto be constant, that is, 1.0 independently of the distance L between thecoordinates of the color difference component values HLCb and HLCr ofthe HL color analyzed from the input image and the HL color blackbodylocus. That is, the reliability R_dis is defined as follows:When LMax>L,R _(—) dis=1.0When LMaxL,R_dis=0.0  (3′)

For example, when it is empirically judged based on the distance betweenthe coordinates of the color difference component values HLCb and HLCrof the HL color analyzed from the input image and the HL color blackbodylocus whether or not an HL color is erroneously detected, the constantreliability shown in FIG. 6B can enhance an effect compared to thatshown in FIG. 6A. For example, when the distance L assumes a smallvalue, the color balance correction can be positively executed. On theother hand, when the distance L assumes a large value, an effect ofstrongly suppressing the color balance correction very much can beexpected.

As yet another method of calculating the reliability R_dis, for example,nonlinear functions may be used, as shown in FIGS. 6C and 6D. In FIGS.6C and 6D, as in FIG. 6A, let L be a distance between the coordinates ofthe color difference component values HLCb and HLCr of the HL coloranalyzed from the input image and the HL color blackbody locus. Also,let R_dis be a reliability, which is calculated based on the distance L,of the color difference component values HLCb and HLCr of the HL coloranalyzed from the input image. Furthermore, Lmax used upon calculationof the reliability R_dis is set in advance. In this embodiment, assumethat the reliability R_dis assumes a value ranging from 0 (minimum) to1.0 (maximum).

In FIG. 6C, as a difference from FIG. 6A, within a range in which thedistance L between the coordinates of the color difference componentvalues HLCb and HLCr of the HL color analyzed from the input image andthe HL color blackbody locus is small (0≦L<L′), the reliability R_dis isset to be constant (1.0). Furthermore, within a range in which thedistance L from the HL color blackbody locus is large (L′≦L<LMax), thereliability R_dis is set to be decreased in proportion to the distance.When L is larger than Lmax, the reliability R_dis is set to be zero. Inthis case, the value L′ may be empirically defined, or may be varied incorrespondence with image characteristics. Therefore, FIG. 6C is definedas follows.When 0≦L<L′,R_dis=1.0When L′≦L<Lmax,R_dis=(LMax−L)/LMaxWhen LMax≦L,R_dis=0.0  (3″)

In FIG. 6D, as a difference from FIG. 6A, positions from L=0 to LMax arecoupled by an S-curve to smoothly change the reliability R_dis. TheS-curve is set to make the reliability R_dis be closer to zero when thedistance L between the coordinates of the color difference componentvalues HLCb and HLCr of the HL color analyzed from the input image andthe HL color blackbody locus is small, and to make the reliability R_disbe closer to 1.0 when the distance L is large.

In both FIGS. 6C and 6D, when the distance L between the coordinates ofthe color difference component values HLCb and HLCr of the HL coloranalyzed from the input image and the HL color blackbody locus assumes asmall value, a high reliability R_dis is set; when the distance Lassumes a large value, a low reliability R_dis is set. Also, thereliability R_dis is changed according to the distance L. Therefore,when the distance L between the coordinates of the color differencecomponent values HLCb and HLCr of the HL color analyzed from the inputimage and the HL color blackbody locus assumes a large value, thecorrection processing can be positively executed. Furthermore, since thereliabilities R_dis of images having slightly different HL colors do nothave a large difference, these images can undergo the correctionprocessing to have nearly the same effects.

Arbitrary conversion methods may be used as long as they are methods ofchanging the HL color based on the distance between the coordinates ofthe color difference component values HLCb and HLCr of the HL coloranalyzed from the input image and the HL color blackbody locus, as shownin FIGS. 6A to 6D. The method of modifying the HL color of the inputimage, which is calculated by the image analysis unit 202, using the HLblackbody locus based on the spectral distribution, which is held inadvance, has been described.

As still another image analysis result modification method, a method ofmodifying the HL color of the input image, which is calculated by theimage analysis unit 202, using a relationship between a human skin colorand the HL color analyzed from the input image may be used in additionto the aforementioned methods. The method of modifying the HL color ofthe input image using a correlation between a human skin color and theHL color analyzed from the input image will be described below.

Using a human skin color as an object color, the object color and HLcolor can be plotted on the Cb-Cr plane on the YCbCr space like anobject color 1201 and HL color 1204 in FIG. 7A under a 3000K lightsource. Also, the object color and HL color can be plotted like anobject color 1203 and HL color 1206 under a daylight light source(5500K), and like an object color 1202 and HL color 1205 under an 7000Klight source.

As can be seen from FIG. 7A, as a characteristic feature, therelationship between the object color and HL color is maintained nearlyconstant even when the light source is changed. As the relationship inthis case, a relational expression between the HL color and object coloris constant. As another characteristic feature, if the object color inthis case is a human skin color, lightness values of skin colors arelargely different depending on races, but color components do not have alarge difference, as disclosed in various references. Therefore, as acharacteristic feature, the relationships between the HL colors and skincolors shown in FIG. 7A are nearly constant according to races.Therefore, as the relationship between the two colors, as shown in FIG.7A, is broken larger, the object color and/or the HL color are/is morelikely to be erroneously detected.

Using these characteristic features, the object color and HL color canbe expressed by a vector relationship. That is, it can also be expressedthat directions and lengths of vectors from the coordinates of HL colorsto those of corresponding object colors on the Cr-Cb plane are the same.Hence, in this embodiment, a vector between a reference object color andreference HL color is formed as a reference vector, and a detectedvector between an object color and HL color is properly modified bymodifying the HL color so as to be closer to the reference vector. Inthis specification, color components corresponding to a human skin colorwill also be described as “skin color” based on the aforementionedreason, for the purpose of convenience.

In order to properly modify a detected vector, a measure of reliabilityis calculated to determine a modification amount. Assume that thereliability becomes lower as the relationship between an object colorand HL color is broken larger than that between the reference objectcolor and reference HL color. Hence, by modifying the positionalrelationship between an object color and HL color according to thereliability, a detected vector can be made closer in a proper direction.

Cases will be described below wherein a detected vector is proper andimproper. In this case, a white point under a daylight light source isset as the reference HL color, and a human skin color under the daylightlight source is set as the reference object color.

This image processing apparatus holds in advance a reference vector1303, which is determined based on a reference object color 1301 andreference HL color 1302, which are set, as shown in FIG. 7B. On theother hand, a detected vector 1306 is calculated from an object color(ObjCb, ObjCr) 1304 and HL color (HLCb, HLCr) 1305 calculated by theobject color calculation unit 303.

When the reference vector 1303 and detected vector 1306 have the samemagnitude, an inner product is calculated based on their directions, anda reliability R_Vec is calculated by:R_Vec=(1+((inner product of reference vector·detectedvector)/((magnitude of reference vector)^2)))/2  (5)where “X^Y” is “X to the Y-th power”.

When the object color 1304 and HL color 1305 are located at thepositions shown in FIG. 7B, since the reference vector 1303 and detectedvector 1306 have the same direction and magnitude, the inner productis 1. The reliability at that time is 1.

A case will be described below with reference to FIG. 7C wherein thedetected vector is improper. That is, a case will be described belowwherein an HL color (HLCb, HLCr) or object color (ObjCb, ObjCr) iserroneously detected. A detected vector 1403 determined by an objectcolor 1401 and HL color 1402 shown in FIG. 7C may have a different angleand magnitude from those of a reference vector 1406 determined by areference object color 1404 and reference HL color 1405.

When the object color 1401 and HL color 1402 are located at positionsshown in FIG. 7C, since the directions of the detected vector 1403 andreference vector 1406 are different through 60°, a reliability R_Vec atthis time is 0.75 according to equation (5) used to calculate areliability.

An example in which the HL color 1402 is modified based on thereliability R_Vec (=0.75) obtained to obtain a proper detected vector1403, as described above, will be described first. Note that thereliability may be equivalent to a weighting coefficient.

In this case, the detected HL color (HLCb, HLCr) 1402 will be referredto as a first HL color (f_HLCb, f_HLCr) hereinafter, and a modified HLcolor will be referred to as a second HL color (s_HLCb, s_HLCr)hereinafter. The second HL color can be calculated by:s _(—) HLCb=b _(—) HLCb+(f _(—) HLCb−b _(—) HLCb)×R_Vecs _(—) HLCr=b _(—) HLCr+(f _(—) HLCr−b _(—) HLCr)×R_Vec  (6)wherein a reference HL color (b_HL) is a point moved from the objectcolor 1401 by a distance corresponding to the reference vector 1406. Thereference HL color (b_HL) is an HL color 1407 in FIG. 7C.

In this case, when the reliability is 1, since it is judged that the HLcolor is reliable, the first HL color is used as the second HL color. Onthe other hand, when the reliability is 0, since it is judged that theHL color is not reliable, the reference HL color is used as the secondHL color. When the reliability is larger than 0 and is smaller than 1,the second HL color is determined between the reference HL color andfirst HL color according to a weighting coefficient.

The range and value of the reliability, reference HL color, andreference object color can be arbitrarily determined. Also, apredetermined color plane or space may be, for example, any of the YCbCrspace, YC1C2 space, and CIE L*a*b* space. In the reliability calculationmethod of the present invention, the second HL color calculation methodis not limited to equations (6) described above. The HL colorinformation and object color information of the image modified by theaforementioned processing are input to the light source estimation unit204 as the subsequent stage.

(Light Source Estimation Unit 204)

Details of the light source estimation unit 204 will be described belowwith reference to the flowchart shown in FIG. 8. In the presentinvention, a method of estimating the type or color temperature of alight source at the time of shooting from an input image is notparticularly limited. In this case, processing will be described byapplying a technique disclosed in Japanese Patent Laid-Open No.2002-152772 as an example. This processing is implemented when the CPU101 reads out and executes a program stored in, for example, the ROM 103as a storage unit included in the image processing apparatus 100 whichcan implement the present invention.

Referring to FIG. 8, the light source estimation unit 204 determines agray axis in the input image in step S1501. The gray axis is determined,as shown in FIGS. 9A and 9B, using HLCb and HLCr as the color of theaforementioned highlight region.

As shown in FIG. 9A, a straight line 1601 perpendicular to the colorplane Cb-Cr on the YCbCr space is set as the gray axis. Alternatively,as shown in FIG. 9B, a straight line 1701 which satisfies (Cb=HLCb,Cr=HLCr) at Y=255 and (Cb=0, Cr=0) at Y=0 may be set as the gray axis.

Normally, a dark part of an image generated by an image capturingapparatus becomes dark since light does not reach a light-receivingelement of the image capturing apparatus. Therefore, the dark part ofthe image is insusceptible to a light source. Hence, depending on thecharacteristics of the image capturing apparatus, when the latter grayaxis is set, as shown in FIG. 9B, the light source estimation precisionis often improved.

In step S1502, the light source estimation unit 204 extracts objectcolor pixels and gray pixels in the input image to generate a map. Themap is generated according to the sequence shown in FIG. 10. Note that atarget image of the light source estimate may be the input image.However, since the input image has a high resolution, as describedabove, this embodiment will be described under the assumption that thelight source estimation is executed using the reduced image generatedfor the histogram calculation processing. As a result, the processingtime required for the light source estimation can be shortened, and thereduced image, which has already been generated, can be commonly usedfor different processes, thus saving a time required to generate areduced image again.

The map generation processing shown in FIG. 10 will be described below.Assume that a gray/skin color map MAP[y][x] is defined as array data,and is held in, for example, the RAM 102. In step S1801, the lightsource estimation unit 204 initializes respective elements of thegray/skin color map MAP[y][x] to zero. In step S1802, the light sourceestimation unit 204 converts RGB component values of a pixel of interestof the reduced image to be processed into YCbCr values. In step S1803,the light source estimation unit 204 refers to the gray axis set on theYCbCr space in FIG. 9A or 9B to calculate a distance between the YCbCrvalues of the pixel of interest and the gray axis according to a knownmethod of calculating a distance between a point and straight line. Ifthe calculated distance is smaller than a predetermined threshold, thelight source estimation unit 204 determines that the pixel of interestis a gray axis adjacent pixel (YES in step S1803), and sets a memberassociated with the position of the pixel of interest in thetwo-dimensional array MAP[y][x], which is prepared to have the samewidth and height as those of the reduced image, to be 1 in step S1804.

If it is determined that the pixel of interest is not a gray axisadjacent pixel (NO in step S1803), the light source estimation unit 204checks in step S1805 if the pixel of interest is a skin color adjacentpixel. In this embodiment, a cylindrical object color region 1901 shownin FIG. 11 is set on the YCbCr space based on the object colorinformation (ObjY, ObjCb, ObjCr) calculated by the object colorcalculation unit 303. This region will be referred to as a skin colorregion hereinafter. Referring to FIG. 11, the object color region 1901is defined by a cylinder which has the object color (ObjCb, ObjCr) asthe center on the Cb-Cr color plane as the center, and a radius ObjR,which assumes a predetermined value. A height of the cylinder, which isdefined by ObjYmax and ObjYmin, can be calculated, for example, by:ObjYmax=ObjY+ΔhObjYmin=ObjY−Δh  (7)where Δh is a predetermined constant, and ObjY is a luminance componentof the object color calculated by the object color calculation unit 303.

When the pixel of interest is included in the object color region 1901,the light source estimation unit 204 judges in step S1805 that the pixelof interest is located adjacent to the object color (YES in step S1805),and sets a member associated with the position of the pixel of interestin MAP[y][x] to be 2 in step S1806. If the processing is complete forall pixels in the input image (YES in step S1807), this processingsequence ends. Note that this embodiment has explained the skin colorregion to have the cylindrical shape shown in FIG. 11 for the sake ofsimplicity. However, the present invention is not limited to this. Forexample, the skin color region may have a spherical shape which has thecalculated object color as the center and has the radius ObjR, or othersimilar shapes. Even in this case, such modifications are incorporatedin the scope of the present invention.

When the aforementioned gray/skin color pixel map can be successfullygenerated using FIG. 10, the light source estimation unit 204 thenestimates a light source at the time of shooting in steps S1503 to S1507in FIG. 8. The light source estimation method is not particularlylimited in the present invention, but the following description will begiven using the method described in Japanese Patent Laid-Open No.2002-152772. According to Japanese Patent Laid-Open No. 2002-152772, acolor balance correction of a normal camera is attained by multiplyingRGB components by adjustment gains. Image data obtained by suchadjustment shows a behavior different from image data obtained fromspectral distribution characteristics of the camera.

More specifically, assume that loci of gray and skin color pixelscalculated from the spectral characteristics of a camera under variouslight sources are used as blackbody loci. Inside the camera, the grayand skin color pixels before the color balance correction aredistributed on the blackbody locus. However, when RGB components aremultiplied by predetermined gains to attain the color balance correctioninside the camera, the distribution of the gray and skin color pixels ofthe image obtained as a result of such color balance correction does notmatch the blackbody locus.

Using such characteristics, the image that has undergone the colorbalance correction is multiplied by various coefficients for respectivepixels to attain inverse conversions of the color balance correction.Respective pixels of the image after each inverse conversion are plottedon a blackbody locus plane, and the number of gray and skin color pixelsadjacent to the blackbody locus is counted. A case corresponding to themaximum counted number of pixels corresponds to a state at the time ofshooting before the color balance correction, and a light source at thetime of shooting is determined based on the blackbody locuscorresponding to the distribution including the largest number of grayand skin color pixels. For further details of the light sourceestimation method, please refer to Japanese Patent Laid-Open No.2002-152772.

The processes in steps S1503 to S1507 will be described below withreference to Japanese Patent Laid-Open No. 2002-152772. In step S1503,the light source estimation unit 204 multiplies respective RGB colorcomponents of a pixel of interest in the reduced image by variousinverse conversion coefficients to attain inverse conversions of thecolor balance correction inside the camera, as described above.

In step S1503, the light source estimation unit 204 initially determinescoefficients r, g, and b required to attain an inverse conversion.Assume that the coefficients r, g, and b are those which vary within,for example, a range from 0.5 to 1.5, and are incremented by 0.1. Whenthe method described in Japanese Patent Laid-Open No. 2002-152772 isused intact, the coefficients r, g, and b have to be used in 10³, thatis, 1000 inverse conversions. When the 1000 inverse conversions areapplied to all target pixels included in the gray/skin color region, ahuge amount of processing is required, posing a problem.

Hence, in consideration of this problem, this embodiment skips inverseconversion processes in directions opposite to the HL color of an image.More specifically, the HL color values (HLY, HLCb, HLCr) of the imageanalyzed by the image analysis unit 202 are converted into values on anRGB color space to obtain (HLR, HLG, HLB) values. HLR and HLB arecompared, and when HLR>HLB, the subsequent inverse conversion processingis executed only when r>b of combinations of the inverse conversioncoefficients r, g, and b. Conversely, when HLR<HLB, the subsequentinverse conversion processing is executed only when r<b of combinationsof the inverse conversion coefficients r, g, and b.

As described above, the HL color of the input image reflects the type ofa light source in a shooting environment to some extent at a very highprobability even when the input image has undergone the color balancecorrection inside an image capturing apparatus. For example, in an imageshot under an incandescent light source of around 3000K, the HL color ofthe image includes color components in an orange direction in place of ablue direction in most cases even when the color balance correction hasbeen applied in the image capturing apparatus. Conversely, in an imageshot under a high color temperature situation such as under shadeoutdoors in fair weather, the HL color of the image includes colorcomponents in the blue direction in most cases even when the colorbalance correction has been applied in the image capturing apparatus.

Therefore, by limiting the coefficients used to perform inverseconversions according to the balance of the color components of the HLcolor of the image, calculations which are not related to the image canbe excluded. A light source estimation color temperature obtained as aresult of the processing does not assume a calculated value opposite tothe HL color of the image, thus allowing a stable color balancecorrection.

In step S1503, the light source estimation unit 204 multiplies RGBcomponents of the pixel of interest of the reduced image by the inverseconversion coefficients r, g, and b determined by the aforementionedprocessing to obtain R′, G′, and B′. R′, G′, and B′ are respectivelycalculated by:R′=R×rG′=G×gB′=B×b  (8)

However, in this embodiment, the inverse conversion is executed onlywhen it is judged in step S1503 with reference to the aforementionedgray/skin color pixel map that the pixel of interest is a gray or skincolor pixel, unlike the Japanese Patent Laid-Open No. 2002-152772. Inthis case, let count_g be the number of pixels judged as gray pixels,and count_h be the number of pixels judged as skin color pixels.

In this way, compared to Japanese Patent Laid-Open No. 2002-152772 thatuses all pixels in an image, the method of this embodiment executes thelight source estimation by excluding pixels which are not related togray/skin color pixels, thereby improving the light source estimationprecision and shortening the processing time required for the lightsource estimation.

After that, in step S1504 the light source estimation unit 204 convertsR′G′B′ components into predetermined color components, and calculates adistance from a blackbody locus defined in advance on a predeterminedcolor plane. If the distance is equal to or smaller than a predeterminedvalue, the light source estimation unit 204 judges that the R′G′B′values after the inverse conversion are located at a position adjacentto the blackbody locus, and counts up the number of pixels.

Since the processes in steps S1503 and S1504 are executed for all pixelsof the input reduced image, the number of pixels adjacent to theblackbody locus can be calculated in association with one set of inverseconversion coefficient values (r, g, b). Then, the processes in stepsS1503 to S1505 are repetitively executed for various inverse conversionsets, which are prepared in advance (S1506).

As a result, the light source estimation unit 204 calculates and storesan inverse conversion set (r_max, g_max, b_max) which corresponds to themaximum number of pixels adjacent to the blackbody locus in step S1507.Furthermore, the light source estimation unit 204 determines a colortemperature K of a light source at the time of shooting according to thedistribution of skin color/gray pixels on the blackbody locus in thatset (please refer to Japanese Patent Laid-Open No. 2002-152772 fordetails). In this case, let count_max be a maximum value of the numberof pixels adjacent to the blackbody locus.

A unit of K in this case is “Kelvin”, and a value ranging from about2500 to 10000 is normally held. Normally, when a light source at thetime of shooting corresponds to daylight, a value around 5500K is held.In case of an orange light source such as an incandescent lamp, a valuearound 3000K is held. Also, in case of a cloudy day or shade, a valuearound 8000K is held.

In this case, a light source estimation reliability R_lse can be definedby:R _(—) lse=(count_max)/(count_(—) g+count_(—) h)  (9)

In equation (9), when count_max=(count_g+count_h), R_lse=1.0. This meansthat all pixels obtained when a pixel of interest is judged as a skincolor or gray pixel in step S1503 are included in a range adjacent tothe blackbody locus, and it can be judged that a light source estimationvalue as a result of the processing has a very high reliability. Thelight source estimation unit 204 has been described.

(First Color Conversion Unit 205)

The first color conversion unit 205 shown in FIG. 2A will be describedbelow. The first color conversion unit 205 executes first colorconversion processing for the purpose of canceling the color balancecorrection applied inside a camera. In this embodiment, the first colorconversion processing can be executed by multiplying RGB componentvalues of respective pixels in the input image by the stored inverseconversion set (r_max, g_max, b_max).

(Second Color Conversion Unit 206)

The second color conversion unit 206 shown in FIG. 2A executes secondcolor conversion processing. The purpose of the second color conversionprocessing is to apply a color balance correction that considers humanvisual characteristics to an image, which has undergone the first colorconversion processing to cancel the color balance correction inside acamera, based on the estimated color temperature of a light source atthe time of shooting.

As one example of such color balance correction, the followingdescription will be given using CIECAM02. The CIECAM02 is a ColorAppearance Model formulated by CIE in 2002. The CIECAM02 allows to setan adaptation factor D of the human visual characteristics in additionto input parameters associated with viewing conditions (c, Nc, F, whitepoint XYZ values of a viewing light source, etc.)

Using these parameters, those corresponding to a viewing condition A areset, and an input image is converted from an XYZ color space into aviewing condition independent color space JCh. From the converted JChcolor space, another viewing condition B is designated to convert theimage onto the XYZ space, thus reproducing the same “appearance” underboth the viewing conditions.

The CIECAM02 is disclosed in various references: for example, CIEPublication, TC8-01, A Color Appearance Model for Colour ManagementSystems. For this reason, such references are to be referred to fordetails, and this embodiment will explain only an overview. Thisembodiment will explain a method of reproducing “appearance” under alight source at the time of shooting as that under a predeterminedviewing condition (for example, a monitor or printed matter) using theCIECAM02.

FIG. 12 shows the processing sequence by the second color conversionunit 206. The following description will be given along this sequence.Note that this processing is implemented when the CPU 101 reads out andexecutes a program stored in, for example, the RAM 102 as a storage unitincluded in the image processing apparatus 100 that can implement thepresent invention.

Referring to FIG. 12, the second color conversion unit 206 calculatesvarious parameters according to the color temperature of a light sourceat the time of shooting, which is estimated by the light sourceestimation unit 204, in step S2001. The parameters mainly include whitepoint XYZ values and an adaptation factor Din under the light source atthe time of shooting.

The white point XYZ values are calculated by the following method. Thatis, as shown in FIG. 13, YCbCr values at principal color temperaturesare stored in advance in a storage unit such as the ROM 103.Furthermore, an interpolation operation is made according to theestimated color temperature to calculate a white point (EsY, EsCb, EsCr)corresponding to the color temperature obtained as a result of the lightsource estimation. For example, when the estimated color temperature is3750K, since it is intermediate between 2000K and 5500K, white pointvalues (EsY, EsCb, EsCr) are (255, −15, 15).

The calculated white point values (EsY, EsCb, EsCr) can be convertedinto XYZ values via sRGB space by known formulas. In step S2001, thesecond color conversion unit 206 also determines the adaptation factorDin under the light source at the time of shooting. The adaptationfactor Din may be set to be a pre-set fixed value (for example,Din=0.8). With the processes executed so far, a highlight color based onthe estimated light source at the time of shooting is calculated. Basedon this calculation result, a first highlight color is determined.

The calculation method of parameters under the estimated light source atthe time of shooting has been described. As a point to be considered inthis case, the light source estimation result is not always correct. Forexample, the white point values (EsY, EsCb, EsCr) calculated based onthe light source estimation result assume nearly the same values as theHL color (HLY, HLCb, HLCr) calculated by the image analysis unit 202. Inthis case, since it is determined that the reference white point iscorrect, the color balance correction can be executed. However, whenthese values largely deviate from each other, it has to be judged whichwhite point is to be finally used to execute the correction processing.

In consideration of the above problem, this embodiment will explain amethod of calculating a final reference white point using both the HLcolor obtained by the image analysis unit 202 and the white pointobtained as a result of the light source estimation in step S2002 inFIG. 12.

FIG. 14A depicts the final white point determination method in thisembodiment. FIG. 14A shows the Cb-Cr color plane, and also a white point2201 calculated from the light source estimation result, an HL color2202 calculated by the image analysis unit 202, and a final white point2203 calculated from both the white point 2201 and HL color 2202. Inthis embodiment, as a simplest example, a final white point (FinY,FinCb, FinCr) is calculated by:FinY=(EsY+HLY)/2FinCb=(EsCb+HLCb)/2FinCr=(EsCr+HLCr)/2  (10)where (EsCb, EsCr) are the coordinates of the white point 2201, and(HLCb, HLCr) are those of the HL color 2202.

Note that the white point 2201 calculated from the light sourceestimation result corresponds to a first HL color, the HL color 2202calculated by the image analysis corresponds to a second HL color, andthe final white point 2203 which is finally calculated corresponds to athird HL color.

Equations (10) mean that the final white point is set at a middle pointbetween the white point and HL color. The final white point values areconverted into XYZ values by known formulas, and the color balancecorrection (to be described later) is executed using the XYZ values.Using this method, third highlight color calculation processing isexecuted.

When the color balance correction is executed using the final whitepoint calculated by the above method, a serious adverse effect can besuppressed compared to a case in which the color balance correction isexecuted using either one of the white point and HL color, thus allowingthe stable color balance correction.

Note that in equations (10), the final white point 2203 is set at themiddle point between the white point 2201 and HL color 2202 for the sakeof simplicity. However, this embodiment is not limited to this. Forexample, the coordinates of the final white point may be determinedusing fixed internal division ratios W_a and W_b, which are set inadvance, as described by:FinY=EsY×W _(—) a+HLY×W _(—) bFinCb=EsCb×W _(—) a+HLCb×W _(—) bFinCr=EsCr×W _(—) a+HLCr×W _(—) b  (11)

When the following constraint condition is used in equations (11), thefinal white point exists on two line segments.

Constraint condition: W_a≧0, W_b≧0, and W_a+W_b=1.0

For example, when the processing precision in the light sourceestimation of the light source estimation unit 204 is relativelyreliable, the internal division ratio values are respectively set to beW_a=0.6 and W_b=0.4 to increase a weighting coefficient of the (EsY,EsCb, EsCr) values, so as to determine the final white point.Conversely, when the performance of the light source estimation unit 204is low, the weighting coefficient of the (EsY, EsCb, EsCr) values may bedecreased. That is, the weighting coefficient ratio can be changedaccording to the processing in the light source estimation of the lightsource estimation unit 204.

Also, in equations (11), the value (W_a+W_b) need not always be set tobe 1.0 as long as it falls within a range that satisfies the effect ofthe present invention. For example, a constraint condition that(W_a+W_b) falls within a range from 0.9 to 1.1 may be set. In this case,a trapezoidal range 2301 bounded by a solid line, as shown in FIG. 14B,corresponds to an existence range of the final white point.

When the final white point is determined in this way, a serious adverseeffect can be suppressed compared to a case in which either of (EsY,EsCb, EsCr) and (HLY, HLCb, HLCr) are used, thus allowing the stablecolor balance correction. The final white point calculation method inthis embodiment has been described.

Subsequently, in step S2003 shown in FIG. 12, the second colorconversion unit 206 determines parameters in an viewing environment inwhich the user views the input image. The viewing environment in thiscase includes, for example, the monitor 111. As is well known, a whitepoint of sRGB monitor is set to be D65. That is, in case of the YCbCrspace, this white point corresponds to (255, 0, 0). By similarlyconverting these values into XYZ values, the XYZ values in the viewingenvironment can be obtained. Also, an adaptation factor Dout in theviewing environment can also be determined based on a pre-set fixedvalue as in the above description.

Note that in the above description, white point coordinates underrespective light sources on the YCbCr space are held for the sake ofsimplicity. However, the present invention is not limited to this. Forexample, representative coordinates on an XYZ space, sRGB space, or CIEL*a*b* space may be held. The white point coordinates may be expressedby mathematical expressions on a predetermined color space in place ofholding representative coordinates. The present invention may use eitherof these methods.

In step S2004 in FIG. 12, the second color conversion unit 206 convertsRGB values (on the sRGB color space in this embodiment) of a pixel ofinterest of the input image onto the XYZ color space by known conversionformulas. In step S2005, the second color conversion unit 206 convertsthe XYZ values of the parameters (white point XYZ values, Din) under thelight source at the time of shooting, which are calculated in stepS2001, onto the JCh color space using conversion formulas described in aknown reference of the CIECAM02.

In step S2006, the second color conversion unit 206 converts the JChvalues of the parameters (XYZ values of the final white point, Dout) inthe viewing environment, which are calculated in step S2003, into X′Y′Z′values using conversion formulas described in a known reference of theCIECAM02. Finally, the second color conversion unit 206 converts theX′Y′Z′ values onto the sRGB color space using known conversion formulasto obtain corrected R′G′B′ pixel values in step S2007.

The processes in steps S2004 to S2007 are applied to the entire inputimage while moving the position of the pixel of interest (S2008). Withthe aforementioned processing, a state approximate to “appearance”viewed at the time of shooting can be reproduced on the monitor as theviewing environment.

Features of the image obtained as a result using the aforementionedprocessing will be described below with reference to FIG. 15. In FIG.15, the abscissa plots a color difference, and the ordinate plots aluminance value. White dots indicate the gray axis of the image which isinput to the second color conversion unit 206 and suffers color fog, andblack dots indicate pixel values after the second color conversionprocessing. Since the white dots suffer white fog, they exist atpositions deviating from the luminance axis. As shown in FIG. 15, anamount of change 2402 of pixels on a dark part region side is smallerthan an amount of change 2401 of pixels on a bright part region side.

When the aforementioned second color conversion processing is applied towhite dot pixels, a white dot axis is corrected toward the luminanceaxis, that is, in an achromatic color direction due to a humanadaptation function, and the correction amount becomes larger for thebright part region (higher-luminance pixels) and becomes smaller for thedark part region (lower-luminance pixels).

That is, paying attention to a plurality of pixels having the same huevalues and the same saturation values of input pixel values, an amountof change of the second color conversion becomes larger for brighterpixels than darker pixels. This is based on the human visualcharacteristics in which the adaptation factor for a bright part ishigher than that for a dark part. With this processing, an image moreapproximate to “appearance” at the time of shooting can be reproducedthan the conventional method.

Also, by setting the adaptation factors Din and Dout to be less than1.0, an imperfect adaptation state which is important in the humanvisual characteristics can be considered, and an image more approximateto “appearance” at the time of shooting can be reproduced than theconventional method.

As described above, according to this embodiment, According to thepresent invention, image adverse effects described in the related artcan be suppressed. That is, when a relative positional relationshipbetween a highlight color and object color (for example, human skincolor) extracted from image data deviates from that in a proper case, itis judged that at least one of the highlight color and object color iswrong, and a low reliability is set for the extracted color information.By suppressing a correction amount at the time of the color balancecorrection using this reliability, the image adverse effects describedin the related art can be suppressed. Hence, using the image processingapparatus and method of the present invention, a stable color balancecorrection without any serious adverse effects can be implemented.

Second Embodiment

This embodiment will explain a method of setting a reliability R_Veccalculated by an image analysis result modification unit 203 whilegiving still other examples. As a characteristic feature described inthe first embodiment, a roughly constant relationship between an HLcolor and object color is maintained even when a light source ischanged. The method of setting the reliability R_Vec based on thischaracteristic feature will be described below with reference to FIG.16A.

FIG. 16A shows a Cb-Cr plane of a YCbCr space. A reliability isdetermined to fall within a range from 0 to 1.0. In this case, when anHL color (HLCb, HLCr) is located at coordinates 2502 in the fourthquadrant to have coordinates (ObjCb, ObjCr) of an object color as a skincolor as an origin, since it is judged that the positional relationshipbetween the object color and HL color is reliable, a reliability=1 isset. On the other hand, when the HL color is located at coordinates 2503in the second quadrant, since it is judged that the positionalrelationship between the object color and HL color are not reliable, areliability=0 is set.

When the HL color located at the coordinates 2502 in FIG. 16A moves toform a circle 2504 to have the skin color as the coordinates 2501 of theobject color as the center, a reliability becomes lower in predetermineddecrements as the HL color moves to the first quadrant and further tothe second quadrant. On the other hand, likewise, a reliability becomeshigher in predetermined increments as the HL color moves to the thirdquadrant and further to the fourth quadrant.

In the fourth quadrant, when the HL color indicated by the coordinates2502 in FIG. 16A moves along a straight line 2505 that passes throughthe coordinates 2501 of the object color and the coordinates 2502 of theHL color, a reliability becomes higher in predetermined increments asthe HL color is closer to the coordinates 2501 of the object color.Also, a reliability becomes lower in predetermined decrements as the HLcolor is farther away from the coordinates 2501.

A range and value of the reliability can be arbitrarily determined. Apredetermined color plane or space may be, for example, any of a YCbCrspace, YC1C2 space, and CIE L*a*b* space.

As described above, the reliability can be further defined, and the sameeffects as in the first embodiment can be obtained.

Third Embodiment

The first embodiment has explained the processing executed when an imageanalysis result modification unit 203 erroneously detects a first HLcolor. This embodiment will explain processing executed when the imageanalysis result modification unit 203 erroneously detects a first objectcolor, with reference to FIG. 16B.

In the first embodiment, when a calculated reliability R_Vec is high,and a calculated face detection reliability R_face is low, this meansthat an object color is more likely to be erroneously detected, or an HLcolor and object color are more likely to be erroneously detected.

When a detection error is more likely to occur, a proper light source atthe time of shooting cannot be estimated, and the color balance of imagedata may be largely broken. Hence, an erroneously detected value has tobe modified by weighting it based on the obtained reliability. Thereliability may be equivalent to the weighting coefficient. At thistime, both the object color and HL color may be modified or only theobject color may be modified.

Since a possibility of erroneous detection increases with decreasingreliability R_Vec or face detection reliability R_face, the correctioneffect has to be weakened. In order to weaken the correction effect, itis desired to use an HL color and object color under a daylight lightsource as a white reference upon setting a reference HL color (b_HLCb,b_HLCr) and reference object color (b_ObjCb, b_ObjCr). In thisembodiment, a white point under the daylight light source is set as thereference HL color, and a human skin color under the daylight lightsource is set as the reference object color. However, the reference HLcolor and reference object color may be freely set.

An example in which at least one of an object color and HL color ismodified based on the reliability R_Vec and face detection reliabilityR_face will be described below. As described in the first embodiment aswell, a detected HL color (HLCb, HLCr) will be referred to as a first HLcolor (f_HLCb, f_HLCr), and a modified HL color will be referred to as asecond HL color (s_HLCb, s_HLCr). Likewise, a detected object color(ObjCb, ObjCr) will be referred to as a first object color (f_ObjCb,f_ObjCr), and a modified object color will be referred to as a secondobject color (s_ObjCb, s_ObjCr). The second object color can beexpressed by:s_ObjCb=b_ObjCb+(f_ObjCb−b_ObjCb)×R _(—) Vec×R_faces_ObjCr=b_ObjCr+(f_ObjCr−b_ObjCr)×R _(—) Vec×R_face  (12)The second HL color can be expressed by:s _(—) HLCb=b _(—) HLCb+(f _(—) HLCb−b _(—) HLCb)×R _(—) Vec×R_faces _(—) HLCr=b _(—) HLCr+(f _(—) HLCr−b _(—) HLCr)×R _(—)Vec×R_face  (13)Equations (12) and (13) include the reliability R_Vec and face detectionreliability R_face.

FIG. 16B shows a first object color 2601, first HL color 2602, referenceobject color 2603, and reference HL color 2604 on a Cb-Cr plane of aYCbCr space. In this case, as a result of equations (12) and (13), thesecond object color is located on a straight line 2605 which connectsthe reference object color and first object color, and the second HLcolor is located on a straight line 2606 which connects the reference HLcolor and first HL color.

In this case, a predetermined color plane or space may be, for example,any of a YCbCr space, YC1C2 space, and CIE L*a*b* space. The calculationmethods of the second HL color and second object color in the presentinvention are not limited to the above equations.

Fourth Embodiment

In the first embodiment, an image analysis unit 202 calculates an HLcolor (HLY, HLCb, HLCr) in final white point determination processing instep S2002 in FIG. 12. Then, the first embodiment has explained that afinal white point is set at a middle point between the calculated HLcolor and a white point (EsY, EsCb, EsCr) obtained from a light sourceestimation result. However, the present invention is not limited tothis.

For example, the final white point can be determined using a lightsource estimation reliability R_lse (=0 to 1.0) described in the firstembodiment by:FinY=EsY×R _(—) lse+HLY×(1.0−R _(—) lse)FinCb=EsCb×R _(—) lse+HLCb×(1.0−R _(—) lse)FinCr=EsCr×R _(—) lse+HLCr×(1.0−R _(—) lse)  (14)

When a face is successfully detected, a skin color is estimatedaccurately. In this case, the final white point can also be determinedusing a weighting coefficient W_mix that additionally considers a facedetection reliability R_face.FinY=EsY×W_mix+HLY×(1.0−W_mix)FinCb=EsCb×W_mix+HLCb×(1.0−W_mix)FinCr=EsCr×W_mix+HLCr×(1.0−W_mix)  (15)

In equations (15), the weighting coefficient W_mix is given by:W_mix=(R_face+R _(—) lse)/2.0  (16)

Equation (16) expresses that importance is attached to (EsY, EsCb, EsCr)when both the light source estimation reliability and face detectionreliability are high, and is attached to (HLY, HLCb, HLCr) when eitherreliability is low.

As described above, according to this embodiment, the final white pointis adaptively calculated for each image by weighting using the lightsource estimation reliability and face detection reliability. Thus, whenthe light source estimation reliability and face detection reliabilityare high, importance is consequently attached to (EsY, EsCb, EsCr). As aresult, a highly stable color balance correction which suppressescrucial detection errors can be attained compared to the conventionalmethod.

Fifth Embodiment

In the fourth embodiment, a final white point is determined as aninternally divided point by multiplying (EsY, EsCb, EsCr) and (HLY,HLCb, HLCr) by predetermined weighting coefficients. This embodimentwill explain, as another method, a method of determining, as a finalwhite point, a white point which improves a skin color detected by anobject detection unit 302.

FIG. 17 shows details of final white point determination processing instep S2002 in FIG. 12. The sequence will be described below withreference to FIG. 17. This processing sequence is implemented when a CPU101 reads out and executes a program stored in, for example, a ROM 103as a storage unit included in an image processing apparatus 100 to whichthis embodiment is applicable. In FIG. 17, a second color conversionunit 206 calculates an N-th final white point candidate (CanY(N),CanCb(N), CanCr(N)) in step S2701. A candidate point as a highlightcolor candidate is calculated by:CanY(N)=EsY×W_can(N)+HLY×(1.0−W_can(N))CanCb(N)=EsCb×W_can(N)+HLCb×(1.0−W_can(N))CanCr(N)=EsCr×W_can(N)+HLCr×(1.0−W_can(N))  (17)where W_can(N) is a weighting coefficient which increases from 0.0 to1.0 in increments of 1/N, and the N-th candidate point means that (EsY,EsCb, EsCr) and (HLY, HLCb, HLCr) are internally divided to haveW_can(N) as the weighting coefficient. For example, when N=4, sinceW_can increases in increments of 0.25, candidate points in this caseinclude five candidate points indicated by candidate points 2801 shownin FIG. 18A. In this way, highlight color candidate calculationprocessing is executed.

In steps S2702 and S2703, a first color conversion unit 205 and thesecond color conversion unit 206 apply first and second colorconversions to an object color (ObjY, ObjCb, ObjCr) using the calculatedN-th white point candidate point. These processes implement a candidatecorrection unit as corrections for the object color using respectivehighlight color candidates. Note that the first and second colorconversions are the same as those described in the first embodiment, anda detailed description thereof will not be given.

In step S2702, the first color conversion unit 205 applies the firstcolor conversion to (ObjY, ObjCb, ObjCr) using a white point (CanY(N),CanCb(N), CanCr(N)) as an object color value 2710 to obtain (ObjY′,ObjCb′, ObjCr′). In step S2703, the second color conversion unit 206applies the second color conversion to (ObjY′, ObjCb′, ObjCr′) similarlyusing the N-th white point candidate to obtain (ObjY″, ObjCb″, ObjCr″).Note that upon execution of the second color conversion, luminance andcolor difference signals (YCbCr) are required to be converted into XYZvalues. However, such conversion can be implemented by known formulas,and a detailed description thereof will not be given.

It is checked in step S2704 whether or not the object color (ObjY″,ObjCb″, ObjCr″) after the second color conversion processing is improvedcompared to the object color (ObjY, ObjCb, ObjCr) before the colorconversion. In this embodiment, assume that an image processingapparatus holds, in advance, ideal object color values 2711 as idealobject color information used in the checking processing. FIG. 18B showsthe ideal object color values 2711. FIG. 18B shows the Cb-Cr colorplane, and also that an object color 2901 assumes ideal values (IdeCb,IdeCr). As a characteristic feature described above, when an object is ahuman face color, lightness values of human skin colors are differentdepending on races, but skin colors have small differences depending onraces. Hence, an ideal color can be determined in advance in this way.

In step S2705, distance between the object colors before and after thecolor conversion and the ideal object color are calculated. Thefollowing equations can be used as calculation formulas to be used indistance calculations, D_org is the distance between the object colorbefore the conversion and the ideal object color, and D_n is thedistance between the object color after the color conversion, which iscalculated using the N-th white point candidate, and the ideal objectcolor:D_org=sqrt(SQ(ObjCb−IdeCb)+SQ(ObjCr−IdeCr))D _(—) n=sqrt(SQ(ObjCb″−IdeCb)+SQ(ObjCr″−IdeCr))  (18)where SQ(x) is a square of x, and sqrt(x) is a square root of x.

In step S2706, two checking process are executed using D_org and D_n.The following two conditions are defined.D _(—) n<D_org  Condition 1D_min>D _(—) n  Condition 2where D_min is a minimum value of D_n.

If condition 1 is not satisfied, this means that the distance to theideal object color is increased as a result of the color conversion, andthe N-th white point candidate is excluded from the candidates. Ifcondition 1 is satisfied, a determination is made based on condition 2.If condition 2 is satisfied, in step S2705 the current white pointcandidate can optimize the object color, D_n is set in D_min, and thecurrent white point candidate is set in the final white point (FinY,FinCb, FinCr). That is, this process is defined by:FinY=CanY(N)FinCb=CanCb(N)FinCr=CanCr(N)  (19)

When the processes in steps S2701 to S2706 are executed for all thewhite point candidate points, a white point candidate which can optimizethe object color is consequently stored in (FinY, FinCb, FinCr) as thefinal white point. These processes implement a third highlight colordetermination unit. The processing sequence of FIG. 17 has beendescribed.

As described above, according to this embodiment, upon determining thefinal white point, a candidate point which optimizes a human face colorof a plurality of candidate points obtained from both (EsY, EsCb, EsCr)and (HLY, HLCb, HLCr) is determined as the final white point. As aresult, a highly stable color balance correction which suppressescrucial correction errors can be attained compared to the conventionalmethod.

Note that in this embodiment, only one point on the Cb-Cr color plane isset as the ideal object color for the sake of simplicity. However, thepresent invention is not limited to this. For example, an ideal objectcolor may be set on the Cb-Cr color plane as a region in place of apoint, and it may be checked in step S2704 whether or not the objectcolors after the first and second color conversions approach thatregion. Even this case is incorporated in the scope of the presentinvention.

Sixth Embodiment

In the examples of the embodiments described so far, a color balancecorrection strength is suppressed by modifying an analysis result by animage analysis result modification unit 203. However, the presentinvention is not limited to this. In this embodiment, an analysisreliability calculation unit 3001 is newly added, as shown in the blockdiagram of FIG. 19. A case will be described below wherein the analysisreliability calculation unit 3001 calculates a reliability of theanalysis result, this reliability is used as a correction value in acolor balance correction, and a second color conversion unit 3002changes a color balance correction strength (color conversion ratio).Note that units shown in FIG. 19 except for the analysis reliabilitycalculation unit 3001 and second color conversion unit 3002 perform thesame operations as in the first embodiment, and a detailed descriptionthereof will not be given.

(Analysis Reliability Calculation Unit 3001)

FIG. 20 shows the processing sequence of the analysis reliabilitycalculation unit 3001. This processing sequence is implemented when aCPU 101 reads out and executes a program stored in, for example, a ROM103 as a storage unit included in an image processing apparatus 100which can implement this embodiment. Referring to FIG. 20, the analysisreliability calculation unit 3001 calculates a reliability of an HLcolor analyzed by an image analysis unit 202 in step S3101. In stepS3102, the analysis reliability calculation unit 3001 calculates areliability of a relative positional relationship between the HL colorand an object color (human skin color) analyzed by the image analysisunit 202. FIG. 20 shows that the processes in steps S3101 and S3102 areparallelly processed. However, these processes may be executed in apredetermined order so that one of these processes is executed first,and the other process is then executed. Finally, in step S3103 theanalysis reliability calculation unit 3001 calculates a finalreliability using the reliabilities obtained from both the processes insteps S3101 and S3102.

In this embodiment, the analysis reliability calculation unit 3001calculates a reliability of HL color values HLCb and HLCr of an inputimage, which are calculated by the image analysis unit 202, using an HLcolor blackbody locus based on a spectral distribution, which is held inadvance, in step S3101. Assume that a method of calculating thereliability of color difference component values HLCb and HLCr of the HLcolor of the input image using the HL color blackbody locus in thisembodiment uses that of a reliability R_dis which has been describedusing FIGS. 5 and 6 of the image analysis result modification unit 203.Since the reliability calculation method makes the same operations as inthe first embodiment, a detailed description thereof will not be given.

In step S3102, the analysis reliability calculation unit 3001 calculatesa reliability R_Vec based on a relative positional relationship betweenthe object color and HL color. As the calculation method, a method ofexpressing the relationship between the object color and HL color as avector and calculating the reliability using the vector, and a method ofdetermining the reliability R_Vec based on ratios of a distance andangle of the object color and HL color, as described in the firstembodiment, are available. Also, when the object color is calculatedusing face detection, a method of determining the reliability R_Vec inconsideration of a ratio between a face detection reliability R_face andthe reliability R_Vec, as described in the third embodiment, is alsoavailable.

In step S3103, the analysis reliability calculation unit 3001 calculatesa final reliability R_fin as a final reliability from the reliabilityR_dis calculated in step S3101 and the reliability R_Vec calculated instep S3102. A method of calculating the final reliability R_fin will bedescribed below. In this embodiment, the final reliability R_fin iscalculated by multiplying the reliability R_dis and reliability R_Vec:R_fin=R_dis×R_Vec  (20)

For example, when the reliability R_dis is 0.2, and the reliabilityR_Vec is 0.5, the final reliability R_fin is calculated as follows:R_fin=0.2×0.5=0.1

In this embodiment, the final reliability R_fin is calculated bymultiplying the reliability R_dis and reliability R_Vec. However, thepresent invention is not limited to this. As another method, forexample, a value obtained by adding the reliability R_dis andreliability R_Vec and dividing the sum by 2 is calculated as the finalreliability.R_fin=(R_dis+R_Vec)/2  (21)

When conditions are the same as those of the reliability R_dis andreliability R_Vec in the above example, the final reliability R_fin iscalculated as follows:R_fin=(0.2+0.5)/2=0.35

As still another method, for example, as the calculation method of thefinal reliability R_fin, a value obtained by adding weightedreliabilities and dividing the sum by 2 is calculated as the finalreliability:R_fin=(R_dis×W_dis+R_Vec×W_Vec)/2  (22)

When conditions are the same as those of the reliability R_dis andreliability R_Vec in the above example, and when a weighting coefficientW_dis of the reliability R_dis is 1.0, and a weighting coefficient W_Vecof the reliability R_Vec is 0.2, the final reliability R_fin iscalculated as follows:R_fin=(0.2×1.0+0.5×0.2)/2=0.15

As yet another method, for example, a lower reliability of thereliability R_dis and reliability R_Vec is calculated as the finalreliability.R_fin=(lower one of R_dis and R_Vec)  (23)

When conditions are the same as those of the reliability R_dis andreliability R_Vec in the above example, the final reliability R_fin iscalculated as follows:Final reliability R_fin=0.2

As described above, the analysis reliability calculation unit 3001calculates the final reliability R_fin using the reliability R_dis ofthe HL color and reliability R_Vec of the relative positionalrelationship between the skin color and HL color.

(Second Color Conversion Unit 3002)

The final reliability R_fin calculated by the aforementioned processingis input to the second color conversion unit 3002, which suppresses thecolor balance correction strength according to the reliability. In thecolor balance correction, the second color conversion unit 3002calculates (Y, Cb, Cr) values of a reference white point based on acolor temperature value obtained as a result of the light sourceestimation with respect to respective pixel values (R, G, B) of an inputimage, as described in the first embodiment. Then, the second colorconversion unit 3002 applies CIECAM02 based on the white pointinformation to obtain (R′, G′, B′) values after the color balancecorrection.

In this embodiment, at the time of the color balance correction, finalwhite point values (Y_fin, Cb_fin, Cr_fin) are calculated using thefinal reliability R_fin as a weighting coefficient for the white pointinformation. Then, the color balance correction is executed using thefinal white point. A method of controlling this correction amount willbe described below. In this embodiment, the final white point iscalculated by:Y_fin=255−(255−Y)×R_finCb_fin=Cb×R_finCr_fin=Cr×R_fin  (24)

In equations (24), when R_fin=1.0, the final white point is the whitepoint itself obtained as a result of the light source estimation. On theother hand, when R_fin=0.0, the final white point is (Y, Cb, Cr)=(255,0, 0). In the latter case, the color balance correction does not operatein practice, and the correction amount becomes zero in effect.

In this manner, by controlling the reference white point of the colorbalance correction using the final reliability R_fin calculated by theanalysis reliability calculation unit 3001, the correction amount can besuppressed when the final reliability R_fin is low.

In the present invention, by applying R_fin as a weighting coefficientto values before and after the color balance correction in place of thewhite point, the correction amount may be suppressed. In this case,output values (R″, G″, B″) after suppression are obtained by:R″=R′−(R′−R)×R_finG″=G′−(G′−G)×R_finB″=B′−(B′−B)×R_fin  (25)

Seventh Embodiment

In the examples of the embodiments described so far, first colorconversion processing is applied to an input image, and second colorconversion processing is then applied, as shown in FIG. 2A. However,when different processes are connected in series, and each processexecutes complicated arithmetic processing for respective pixels, a longprocessing time is required.

Hence, this embodiment proposes that the processes are executed by thearrangement shown in the block diagram of FIG. 21. That is, thisembodiment proposes a method in which parameters required for the firstand second color conversions are calculated in advance, athree-dimensional lookup table (to be abbreviated as a 3DLUThereinafter) which reflects these parameters is calculated, and the3DLUT is applied to an input image using interpolation calculations.Note that the functional blocks shown in FIG. 21 are the same as thecontents described in the first to sixth embodiments except forcomponents 3201 to 3204, and a description thereof will not be repeated.

A first color conversion parameter calculation unit 3201 determinesimage processing parameters which are the same as those described in thefirst embodiment, and are required to cancel a WB correction inside acamera. According to the first embodiment, an inverse conversion set(r_max, g_max, b_max) corresponding to the maximum number of gray/skincolor pixels adjacent to a blackbody locus calculated by a light sourceestimation unit 204 is determined as the image processing parametersintact, and the determined parameters are input to a 3DLUT generationunit 3203.

Next, a second color conversion parameter calculation unit 3202determines parameters which are the same as those described in the firstembodiment and are required for the second color conversion. Theparameters required for the second color conversion include thoseassociated with a light source at the time of shooting (white point XYZvalues at an estimated color temperature, and an adaptation factor Din)and those associated with a viewing environment (white point XYZ valuesin the viewing environment, and an adaptation factor Dout). Theseparameters are calculated by the method of controlling a reference whitepoint using a final reliability R_fin calculated by an analysisreliability calculation unit 3001 in the same manner as those describedin the sixth embodiment, and are input to the 3DLUT generation unit3203.

The processing of the 3DLUT generation unit 3203 will be described belowwith reference to the flowchart shown in FIG. 22. Note that thisembodiment generates a 3DLUT including RGB components for 17 grids onsRGB space (the total number of grids is 17^3=4913 grids) for the sakeof simplicity. However, the present invention is not limited to this.This processing sequence is implemented when a CPU 101 reads out andexecutes a program stored in, for example, a ROM 103 as a storage unitincluded in an image processing apparatus 100 which can implement thisembodiment.

Referring to FIG. 22, the 3DLUT generation unit 3203 initializesrespective grids first in step S3301. That is, the 3DLUT generation unit3203 resets respective R, G, and B values to zero. In step S3302, the3DLUT generation unit 3203 executes the first color conversion usingfirst color conversion parameters 3310. More specifically, the 3DLUTgeneration unit 3203 calculates R′G′B′ values after the first colorconversion using:R′=R×r_maxG′=G×g_maxB′=B×b_max  (26)

Next, the 3DLUT generation unit 3203 converts the calculated R′G′B′values into XYZ values using known conversion formula in step S3303. The3DLUT generation unit 3203 converts the XYZ values into values on a JChspace using CIECAM02 in step S3304, as described in the firstembodiment, and inversely converts the JCh values into values on an XYZspace in step S3305. Second color conversion parameters 3311 are used inthis case.

In step S3306, the 3DLUT generation unit 3203 converts the XYZ valuesafter the second color conversion into values on the sRGB color space,and stores these values in the 3DLUT as an output value of a currentgrid of interest. The 3DLUT generation unit 3203 repeats the aboveprocesses for all the grids of RGB components, and stores output valuesof the respective grids in the 3DLUT.

Upon completion of the processes in steps S3302 to S3306 (NO in stepS3307), the 3DLUT generation unit 3203 increments a B component value by16 in step S3308. If the B component value exceeds 255, the 3DLUTgeneration unit 3203 increments a G component value by 16, and resetsthe B component value to zero, thus continuing the processing.Furthermore, if the G component value exceeds 255, the 3DLUT generationunit 3203 increments an R component value by 16, and resets the G and Bcomponent values to zero. In case of the 17-grid 3DLUT, theaforementioned processing is repeated 4913 times. When all the RGBcomponent values exceed 255, the 3DLUT generation unit 3203 determinesthat the calculation processing of output values for all the grids iscomplete (YES in step S3307), thus ending the processing.

The 3DLUT generated by the above processing is input to an interpolationunit 3204, which can obtain a corrected image using a knowninterpolation method (a detailed description of which will not be given)for an input image. The 3DLUT is held in a storage unit such as a RAM102, and is read out as needed when it is used. As for the interpolationmethod, various methods such as tetrahedral interpolation and cubicinterpolation are available. In the present invention, any of thesemethods may be used.

As described above, according to this embodiment, output values arecalculated using the first and second color conversion parameters, whichare calculated in advance, for limited grid positions on the sRGB colorspace, and are held as the 3DLUT. By converting an input image byinterpolation calculations using the 3DLUT, an output image can beobtained within a very shorter processing time than that of complicatedcolor conversion processing for respective pixels, especially when aninput image resolution is high.

Note that the first color conversion parameter calculation unit 3201 ofthis embodiment may execute processing using a shift vector described inthe third embodiment. That is, the shift vector value is input to the3DLUT generation unit 3203, which implements color shift processing instep S3302 in the 3DLUT generation unit 3203 by the same method as inthe third embodiment.

This embodiment has described the processing for generating the 3DLUTpremised on three RGB components for the sake of simplicity. However,the present invention is not limited to this. For example, according tothis embodiment, if an input image format includes four CMYK components,processing may be executed using a four-dimensional LUT. That is,processing executed using the multi-dimensional LUT is also included inthe scope of the present invention.

As another embodiment, as shown in FIG. 23, a reliability calculated bythe analysis reliability calculation unit 3001 is input to the 3DLUTgeneration unit 3203. The 3DLUT generation unit 3203 controls correctionamounts of the color balance correction of respective grids of the3DLUT, thus implementing the present invention.

Other Embodiments

The first embodiment has explained the two methods of modifying imagefeature amounts calculated by the image analysis unit 202, that is, themethod of modifying an HL color based on the distance between theblackbody locus and HL color, and the method of modifying one or both ofan object color and the HL color based on the relative relationshipbetween the object color and HL color. However, the present inventionneed not always use both of these methods. Even when one of thesemethods is used, such modification is incorporated within the scope ofthe present invention.

In the method described in the sixth embodiment, the analysisreliability calculation unit 3001 calculates the two reliabilities R_Vecand R_dis, and then calculates the final reliability R_fin using the tworeliabilities. However, the present invention is not limited to this.The final reliability R_fin may be determined based on either value, andsuch modification is included within the scope of the present invention.

The color balance correction in the second color conversion unit 206described in the above embodiment has been described using the CIECAM02.However, the present invention is not limited to this. For example,other color balance correction methods may be used as long as they areexecuted using the HL color of an image or reference white point.

An input image is not limited to an image shot by a digital camera. Animage obtained by converting a digital image input from, for example, adigital camera into that having a display or print resolution may beused as an input image to execute the above processing.

In the above description, the sRGB color space is used as a work colorspace of an input image and processing. However, the present inventionis not limited to this. For example, an AdobeRGB (trademark of AdobeSystems, U.S.A.) color space may be used as a work color space, or othercolor spaces such as a CIE L*a*b* space may be used.

The object detection has been described using a human face region.However, the present invention is not limited to this. For example,objects to be detected may be animals, buildings, and the like, and suchmodification is included within the scope of the present invention.

The present invention may be applied to either a system configured by aplurality of devices (for example, a host computer, interface device,reader, and printer), or an apparatus including a single device (forexample, a printer, copying machine, or facsimile apparatus).

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (for example, computer-readable medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2010-162216, filed Jul. 16, 2010, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: a firstacquisition unit which acquires a color of an object in image data; asecond acquisition unit which acquires a highlight color in the imagedata based on an analysis result of luminance components in the imagedata; and a correction unit which corrects, in a case where a distancebetween the highlight color and the color of the object is equal to orlarger than a predetermined distance on a color plane, the highlightcolor in the image data so that a correction amount of the highlightcolor becomes larger as the distance becomes longer.
 2. The apparatusaccording to claim 1 further comprising: a color balance correction unitwhich applies a color balance correction to the image data using thehighlight color corrected by the correction unit.
 3. A non-transitorycomputer-readable medium storing a program for controlling a computer tofunction as an image processing apparatus according to claims
 1. 4. Theapparatus according to claim 1, further comprising: an estimation unitwhich estimates a light environment at the time of shooting of the imagedata based on the highlight color corrected by the correction unit. 5.The apparatus according to claim 1, wherein the correction unit correctsthe highlight color so that a relative position between the highlightcolor and the color of the object comes close to a relative positionbetween a reference highlight color and a reference object color on thecolor space.
 6. The apparatus according to claim 5, wherein thereference highlight color and the reference object color are a highlightcolor and a color of an object under a predetermined light source. 7.The apparatus according to claim 1, wherein the color of the object is askin color of a face in the image data.
 8. The apparatus according toclaim 1, wherein the color plane is a Cb-Cr plane of a YCbCr space. 9.An image processing method comprising: a first acquisition step ofacquiring a color of an object in image data; a second acquisition stepof acquiring a highlight color in the image data based on an analysisresult of luminance components in the image data; and a correction stepof correcting, in a case where a distance between the highlight colorand the color of the object is equal to or larger than a predetermineddistance on a color plane, the highlight color in the image data so thata correction amount of the highlight color becomes larger as thedistance becomes longer.
 10. An image processing apparatus comprising: acalculation unit which calculates an object color of an object includedin the image data; a detection unit which detects a highlight color ofthe image data; a reliability calculation unit which calculates areliability for a value of the object color calculated by saidcalculation unit and a reliability for a value of the highlight coloracquired by said acquisition unit based on a relative positionalrelationship between the object color and the highlight color on a colorspace, and differences between the object color and the highlight color,and reference values; and a color balance correction unit which appliesthe color balance correction to the object color and the highlight colorby controlling a correction amount for the object color and thehighlight color using the reliability for the value of the object colorand the reliability for the value of the highlight color.
 11. Theapparatus according to claim 10, wherein said color balance correctionunit further comprises: a light source estimate unit which estimates acolor temperature of a light source at the time of shooting of the imagedata; and a color conversion unit which executes a color conversion ofthe image data based on the color temperature estimated by said lightsource estimation unit.
 12. The apparatus according to claim 11, whereinsaid light source estimation unit further comprises a modification unitwhich modifies a highlight color of the image data at the estimatedcolor temperature based on the reliability for the value of the objectcolor and the reliability for the value of the highlight color.
 13. Theapparatus according to claim 11, wherein said color conversion unitsuppresses a color conversion ratio in the color conversion as thereliabilities are lower.
 14. The apparatus according to claim 11,wherein said light source estimation unit further comprises: a settingunit which sets a gray axis on the color space based on the highlightcolor acquired by said acquisition unit; and an estimation unit whichexecutes a light source estimation using pixels adjacent to the grayaxis and pixels adjacent to the object color.
 15. The apparatusaccording to claim 10, wherein said calculation unit further comprises aface detection unit which calculates, as the object color, a skin colorof a human face from the image data.
 16. The apparatus according toclaim 15, wherein said reliability calculation unit calculates thereliability for the value of the object color and the reliability forthe value of the highlight color using a face detection reliabilityobtained by said face detection unit.
 17. An image processing methodcomprising: a calculation step of calculating an object color of anobject included in image data; a detection step of detecting a highlightcolor of the image data; a reliability calculation step of calculating areliability for a value of the object color calculated in thecalculation step and a reliability for a value of the highlight coloracquired in the acquisition step based on a relative positionalrelationship between the object color and the highlight color on a colorspace, and differences between the object color and the highlight color,and reference values; and a color balance correction step of applyingthe color balance correction to the object color and the highlight colorby controlling a correction amount for the object color and thehighlight color using the reliability for the value of the object colorand the reliability for the value of the highlight color.